December 2016 



Applied Research Methods 


A comparison of two methods 
of identifying beating-the-odds 
high schools in Puerto Rico 




REL 


NORTHEAST 
S> ISLANDS 


NATIONAL CENTER for 
EDUCATION EVALUATION 
and REGIONAL ASSISTANCE 


Coby V. Meyers 
University of Virginia 
American Institutes for Research 

Yinmei Wan 

American Institutes for Research 




U.S. Department of Education 

John B. King, Jr., Secretary 

Institute of Education Sciences 

Ruth Neild, Deputy Director for Policy and Research 
Delegated Duties of the Director 

National Center for Education Evaluation and Regional Assistance 

Joy Lesnick, Acting Commissioner 
Amy Johnson, Action Editor 
Elizabeth Eisner, Project Officer 

REL 2017-167 

The National Center for Education Evaluation and Regional Assistance (NCEE) conducts 
unbiased large-scale evaluations of education programs and practices supported by federal 
funds; provides research-based technical assistance to educators and policymakers; and 
supports the synthesis and the widespread dissemination of the results of research and 
evaluation throughout the United States. 

December 2016 

This report was prepared for the Institute of Education Sciences (IES) under Contract 
ED-IES'12'C-0009 by Regional Educational Laboratory Northeast & Islands administered 
by Education Development Center, Inc. The content of the publication does not neces- 
sarily reflect the views or policies of IES or the U.S. Department of Education, nor does 
mention of trade names, commercial products, or organizations imply endorsement by the 
U.S. Government. 

This REL report is in the public domain. While permission to reprint this publication is 
not necessary, it should be cited as: 

Meyers, C. V., & Wan, Y. (2016). A comparison of two methods of identifying beating'the - 
odds high schools in Puerto Rico (REL 2017-167). Washington, DC: U.S. Department of 
Education, Institute of Education Sciences, National Center for Education Evaluation and 
Regional Assistance, Regional Educational Laboratory Northeast Sdslands. Retrieved 
from http://ies.ed.gov/ncee/edlabs. 

This report is available on the Regional Educational Laboratory website at http://ies.ed.gov/ 
ncee/edlabs. 


Summary 


The 2013 National Assessment of Educational Progress results showed that 5 percent of 
grade 8 students in Puerto Rico performed at or above the basic level in math and that 
less than 1 percent scored at or above the proficient level (U.S. Department of Educa- 
tion, n.d. b). Only 62 percent of students who entered public high school in Puerto Rico 
graduated on time in 2011/12, compared with a national average of 82 percent (Stetser & 
Stillwell, 2014). 

Given widespread low academic success, turning around low-performing high schools 
has become one of the top priorities of educators and policymakers in Puerto Rico. One 
response has been the Puerto Rico Research Alliance for Dropout Prevention’s focus on 
reducing the territory’s dropout rate. The alliance members are interested in identifying 
beating-the-odds high schools in Puerto Rico — schools that are performing better than 
expected, given the prior performance and characteristics of the students enrolled — so 
that they can later identify the organizational factors and instructional practices used in 
these schools in order to improve performance in others. 

This study compares two methods of identifying beating-the-odds schools in Puerto Rico. 
It fits with a growing body of research on identifying beating-the-odds schools across 
contexts and through various statistical methods (Abe et al, 2015; Koon, Petscher, & 
Foorman, 2014). The considerations presented here on selecting statistical procedures to 
accurately identify beating-the-odds schools can inform policy work among both Puerto 
Rico stakeholders and a broader national audience. 

School-level data for 2011/12 and 2012/13 from national and territory databases were used 
to identify beating-the-odds public high schools in Puerto Rico from a sample of 159 
schools with at least 40 percent of students from low-income households (high-poverty 
schools). Schools were identified using two methods: a status method that ranked high- 
poverty schools on the basis of their current observed performance and an exceeding 
achievement expectations method that ranked schools on the basis of the extent to which 
their observed performance exceeded (or fell short of) their expected performance. Grad- 
uation rates, reading proficiency rates, and math proficiency rates were analyzed to identify 
schools for each method. The schools identified using each method were then compared 
to determine agreement rates (amount of overlap) in schools identified using each method. 
This report presents descriptive information from comparisons of three groups of schools 
— those identified using the status method, those identified using the exceeding achieve- 
ment expectations method, and all high-poverty public high schools in Puerto Rico. 

The status method identified 17 beating-the-odds schools out of 159 high-poverty schools 
in the analysis sample, and the exceeding achievement expectations method identified 15. 
Six schools were identified using both methods, for a 38 percent agreement rate between 
the two methods. Both groups of beating-the-odds high schools appeared similar to all 
high schools in the sample in average enrollment, student racial/ethnic composition, and 
student-teacher ratio. However, beating-the-odds high schools identified using the status 
method had a lower average percentage of students from households living below the 
poverty line and a lower average percentage of students with disabilities than did beating- 
the-odds high schools identified using the exceeding achievement expectations method 
and all schools in the sample. In addition, some of the schools identified using the status 


method ranked low when the exceeding achievement expectations method was used (and 
when school poverty rate and other student and school characteristics were statistically 
controlled for). 

The analyses suggest that identifying schools with relatively high percentages of stu- 
dents from households living below the poverty line and relatively high performance (the 
status method) is a straightforward method but one that can result in the identification of 
schools that are not achieving at levels higher than expected given their demographics. By 
contrast, the exceeding achievement expectations method can identify schools exceeding 
expectations but with low achievement. The relatively low agreement rate for schools idem 
tified using the two methods illustrates how the methods affect the school identification 
results. Thus, leveraging both methods is a good strategy to identify schools that are both 
performing at higher overall levels and exceeding achievement expectations. 
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Why this study? 


During the past decade states and school districts have focused increasingly on turning 
around persistently low-performing schools (Duke, Tucker, Salmonowicz, Levy, & Saun- 
ders, 2007; Herman et al., 2008; Murphy & Meyers, 2008). As of 2012, there were approx- 
imately 5,000 chronically low-performing schools nationwide that were not providing 
students with an education that prepares them for lifelong success (U.S. Department of 
Education, 2012). Students in these schools have lower scores on assessments and graduate 
at lower rates than students at other schools. At the secondary school level these low-per- 
forming schools are sometimes described as “dropout factories” (Balfanz and Legters, 
2004, 2006). Federal initiatives have demonstrated a commitment to turning around 
those schools, through programs such as School Improvement Grants and Race to the 
Top, which provide funding to support improving academic outcomes, in particular for the 
lowest performing schools (Council of the Great City Schools, 2015). In turn, state edu- 
cation agencies have been under pressure to identify and improve their lowest performing 
schools by developing and implementing supports and interventions to raise academic per- 
formance (Le Floch et al., 2014). Many of these efforts seem to operate under the assump- 
tion that successful supports, initiatives, and interventions in high-performing schools are 
transferable to demographically similar low-performing schools (Murphy & Meyers, 2008). 

The challenge of turning schools around is also prevalent in Puerto Rico, where accord- 
ing to the 2013 National Assessment of Educational Progress, only 5 percent of grade 8 
students performed at or above the basic level in math and less than 1 percent scored 
at or above the proficient level (U.S. Department of Education, n.d. b). Moreover, only 
62 percent of students who entered public high school in Puerto Rico graduated on time 
in 2011/12, compared with a national average of 82 percent (Stetser & Stillwell, 2014). 
Thus, turning around low-performing schools is increasingly a priority for educators and 
policymakers in Puerto Rico. The Puerto Rico Research Alliance for Dropout Prevention 
has been developing a research agenda focused on reducing the territory’s dropout rate. 
As a part of this broad agenda, alliance members partnered with Regional Educational 
Laboratory (REL) Northeast & Islands to identify beating-the-odds high schools in Puerto 
Rico — schools that are performing better than expected given high poverty rates — so that 
they can later identify the organizational factors and instructional practices used in these 
schools. 


This study may be 
useful for Puerto 
Rico stakeholders 
and national 
education leaders 
as they consider 
how to develop and 
establish statistical 
methods that 
reliably identify 
schools that are 
beating the odds 


In alliance meetings, some members of the Puerto Rico Department of Education 
expressed the hope of improving education outcomes by identifying and disseminating 
information on the practices of beating-the-odds schools. The Puerto Rico Department of 
Education does not currently identify beating-the-odds schools. In response to the Puerto 
Rico Research Alliance for Dropout Prevention’s request, REL Northeast & Islands con- 
ducted this study to identify beating-the-odds schools in Puerto Rico using two methods: 
a status method and an exceeding achievement expectations method. The study team 
and members of the Puerto Rico Department of Education agreed that a comparison of 
methods would be useful for the department as it weighs methods of identifying beating- 
the-odds schools and gathering lessons from those schools. Considering the methods by 
which those schools are identified is a critical initial step. 

This study may be useful for Puerto Rico stakeholders and national education leaders as 
they consider how to develop and establish statistical methods that reliably identify schools 
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that are beating the odds. Accurate statistical identification of beating-the-odds schools is 
necessary to ensure that lessons from those schools are worth learning and basing policy 
creation and professional development on. The identification of beating-the-odds schools in 
this report can provide initial information to the Puerto Rico Department of Education as it 
plans how to examine processes and practices of schools that are beating the odds. 

What the study examined 


This report presents the results of analyses that incorporated schooUevel data to identify 
beating-the-odds public high schools in Puerto Rico using two analytic methods and com- 
pares the resulting lists of schools. It then compares the characteristics of schools on both 
lists and of all schools (including beating-the-odds schools) in the sample of high-poverty 
high schools (schools with at least 40 percent of students from low-income households). 

Two research questions guided the study: 

• What is the agreement rate of Puerto Rican public high schools identified as 
beating the odds using the status method and using the exceeding achievement 
expectations method? 

• What are the characteristics of public high schools (for example, poverty rates and 
student-teacher ratios) that are identified as beating the odds using each method? 

Puerto Rican high schools traditionally enroll students in grades 10-12. This study focused 
on the performance of the cohort of students who were expected to graduate in the 2012/13 
school year. Under the status method, schools were ranked on their observed performance 
on two outcome measures: 2012/13 graduation rates and 2011/12 grade 11 combined profi- 
ciency rates on reading and math assessments. Under the exceeding achievement expec- 
tations method, schools were ranked on the basis of the extent to which their observed 
performance on these outcome measures exceeded (or fell short of) their expected perfor- 
mance on the same two outcome measures (see box 1 and appendix A for a description of 
these methods). 

The sample included only high-poverty schools, which includes about 95 percent of public 
schools in Puerto Rico. The schools identified using each method were compared on the 
basis of their agreement rate, defined as “a ratio of the number of schools that appear on 
both sets of the lists to the average number of schools across the two lists” (Abe et al., 2015, 
p. B-7). This report presents descriptive information from comparisons of three school 
groups — those identified using each method and all high-poverty public high schools in 
Puerto Rico (see box 1). 


Under the status 
method, schools 
were ranked on 
their observed 
performance on 
two outcome 
measures: 2012/13 
graduation rates 
and 2011/12 
grade 11 combined 
proficiency rates on 
reading and math 
assessments. Under 
the exceeding 
achievement 
expectations 
method, schools 
were ranked on 
the basis of the 
extent to which 
their observed 
performance on 
these outcome 
measures exceeded 
(or fell short of) 
their expected 
performance on 
the same two 
outcome measures 


What the study found 


The two methods resulted in two different lists of beating-the-odds schools. The ranking 
by status method identified 17 beating-the-odds schools, and the exceeding achievement 
expectations method identified 15 schools; 6 schools were identified using both methods. 
The agreement rate between the two lists of beating-the-odds schools was 38 percent. 
Schools’ rankings on individual measures show varied correlations. 

Beating-the-odds schools are similar to all high-poverty schools in the sample in student 
race/ethnicity composition, average enrollment, and average student-teacher ratio. But 
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Box 1. Data and methods 


Data. This study used school-level data from two sources to identify public high schools in Puerto Rico that were 
beating the odds based on the performance of the cohort of students expected to graduate during the 2012/13 
school year, the most recent cohort for which graduation and achievement data were available at the time of this 
study. Publicly available school-level demographic data were retrieved from the Common Core of Data, maintained 
by the U.S. Department of Education (U.S. Department of Education, 2010, 2013, 2014). Data on school-level 
graduation rates, assessment results, and poverty rates were provided by the Puerto Rico Department of Education. 

Sample. The sample included 159 schools that met all of the following criteria: were identified as regular public high 
schools in the Common Core of Data (vocational schools and alternative schools were not included), had valid data 
on 2011/12 grade 11 reading and math assessment results and 2012/13 graduation rates, were in operation in 
both 2011/12 and 2012/13, and had at least 40 percent of students who were from households living below the 
poverty line (as defined by U.S. Census Bureau, n.d.) in 2009/10. 

Methods of identifying and comparing beating-the-odds schools , The analysis focused on the performance of the 
2012/13 graduation cohort on two outcome measures: the 2012/13 cohort graduation rate and the 2011/12 grade 
11 proficiency rate for reading (in Spanish, the language of instruction in Puerto Rico) and math combined, weighted 
by the number of students tested in each subject. (Weighting is detailed in appendix A.) 

Two methods were used to identify beating-the-odds schools: 

• The status method ranked schools on the basis of their observed performance on each of the outcome mea- 
sures. Schools that ranked among the top 25 percent on both outcome measures were identified as beating- 
the-odds schools. 1 

• The exceeding achievement expectations method estimated schools’ expected performance on the two outcome 
measures (the 2012/13 graduation rate and the 2011/12 grade 11 combined reading and math proficiency 
rate) using ordinary least squares regressions that controlled for grade 8 achievement levels, school poverty 
rate, and other student and school characteristics that might be related to school performance. 

To adjust for grade 8 school achievement levels, possible feeder schools were identified for each high school 
in the sample using information on region, district, school location, and distance between schools. As a proxy for 
prior achievement, the study used the weighted average school-level proficiency rate in grade 8 reading and math 
of feeder schools from 2008/09 — the year the 2012/13 graduating cohort was in grade 8 — as an independent 
variable. The differences between schools’ observed performance and expected performance were calculated, and 
schools were ranked on the basis of differences. Schools that ranked in the top 25 percent on both outcome mea- 
sures were identified as beating-the-odds schools. 

To compare the school identification results, this report presents the two lists of beating-the-odds schools and 
reported rankings for each school on each measure — the 2012/13 graduation rate and the 2011/12 combined pro- 
ficiency rate in reading and math — in both methods. To illustrate why the lists of beating-the-odds schools identified 
using each method do or do not agree, the report presents correlations of the outcomes across methods. Spearman 
rank-order correlation coefficients were calculated between outcome measures across methods. To measure the 
differences and similarities between the two lists of beating-the-odds schools, an agreement rate (the ratio of the 
number of schools that appear on both lists to the average number of schools across the two lists) was computed. 
Finally, the report compares the characteristics of four groups of schools: all beating-the-odds schools identified 
using the status method, all beating-the-odds schools identified using the exceeding achievement expectations 
method, beating-the-odds schools identified using both methods, and all schools in the sample of high-poverty high 
schools (including the beating-the-odds schools). The report presents basic descriptive information for the gradua- 
tion rate outcome, achievement outcome, student demographics, school locale, enrollment, size of teaching staff, 
and student-teacher ratio. 

See appendix A for more details on the data, sample, and methods. 

Note 

1. The 25 percent outpoint for performance measures was an arbitrary decision by the study team. Different numbers of beating-the- 
odds schools would be identified if different outpoints were applied; more stringent criteria would result in fewer identified schools. The 
study team conducted sensitivity analysis using different outpoints for outcome measures (see appendix B). 
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beating-the-odds schools had higher graduation rates and math achievement averages 
than all high-poverty high schools in the sample. However, some beating-the-odds schools 
that were identified using the status method (those that ranked at the top on observed 
performance) ranked low — two were in the bottom 10 percent — when analyzed using 
the exceeding achievement expectations method, which accounted for the demographic 
characteristics of their students. Although a few of the beating-the-odds schools that were 
identified using the exceeding achievement expectations method had lower rankings when 
analyzed using the status method, those differences were not as large. 

The ranking by status method and ranking by exceeding achievement expectations method resulted 
in different lists of beating-the-odds schools 

The status method identified 17 beating-the-odds schools (table 1). The exceeding achieve- 
ment expectations method identified 15 schools. Six schools were identified as beating 
the odds using both methods. Eleven schools were uniquely identified using the status 
method. 1 However, when the exceeding achievement expectations method was applied to 
these schools, their rankings ranged from the 3rd to 97th percentile. This suggests that 
the ranking of schools identified as beating the odds by the status method ranged widely 
after prior performance and demographic differences were statistically controlled for. Some 
of the beating-the-odds schools identified using the status method ranked low on the two 
outcome measures: two schools (schools P and Q) ranked below the 10th percentile on 
the combined reading and math proficiency rate measure; one (school O) ranked in the 
27th percentile on the proficiency rate measure and another school (school I) ranked in 
the 26th percentile on the graduation rate measure. Nine schools were uniquely identi- 
fied using the exceeding achievement expectations method. Their rankings on the two 
outcome measures under the status method ranged from the 34th to the 98th percentile. 

Comparisons of the lists of schools that met each criterion (top 25 percent on the 2012/13 
graduation rate and top 25 percent on the 2011/12 grade 11 proficiency rate) show that the 
agreement rate between the two methods for schools that met the criteria for the same 
measure was 58 percent for the 2012/13 graduation rate measure and 68 percent for the 
2011/12 proficiency rate measure (table 2). But when the lists of schools that met all cri- 
teria for each method (the beating-the-odds schools) are compared, the agreement rate is 
38 percent. The relatively low agreement rate for schools identified using the two methods 
illustrates how the methods — ranking by status versus ranking by exceeding achievement 
expectations — affect the school identification results. 

Comparisons of schools’ rankings on individual measures between methods show low to moderate 
correlations 

For each outcome measure the correlation 2 between schools’ rankings across methods was 
.76 for the graduation rate measure and .51 for the percent proficiency rate (table 3). 

Although schools’ rankings on the graduation rate measure under the two methods tend 
to be related (correlation of .76), schools that ranked high on the proficiency rate outcome 
measure under one method did not necessarily rank high on that measure under the other 
method (.51). The statistical analyses found that the school background factors included in 
the exceeding achievement expectations method together explained higher proportions of 
variances in the proficiency rate outcome measure (36 percent) than in the graduation rate 


The status method 
identified 17 
beating-the- 
odds schools. 

The exceeding 
achievement 
expectations 
method identified 
15 schools. Six 
schools were 
identified as 
beating the 
odds using both 
methods 
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Table 1. Percentile rankings on outcome measures for beating-the-odds high schools in Puerto Rico, 
by identification method, 2011/12 and 2012/13 



Status method 
(percentile ranking 3 ) 

Exceeding achievement expectations method 
(percentile ranking 3 ) 

School 

School 
poverty rate 
(percent) 

Graduation rate, 
2012/13 

Grade 11 combined 
reading and math 
proficiency rate, 
2011/12 

Graduation rate, 
2012/13 

Grade 11 combined 
reading and math 
proficiency rate, 
2011/12 

Identified by both methods 

School A 

82 

99 

92 

99 

96 

School B 

79 

85 

83 

79 

84 

School C 

65 

97 

96 

96 

97 

School D 

69 

97 

91 

98 

92 

School E 

77 

84 

79 

89 

88 

School F 

69 

89 

87 

89 

82 

Identified by the status method only 

School G 

45 

100 

99 

52 

96 

School H 

52 

94 

92 

81 

34 

School 1 

44 

92 

99 

26 

97 

School J 

74 

86 

94 

67 

95 

School K 

52 

92 

97 

65 

94 

School L 

63 

82 

85 

58 

85 

School M 

67 

80 

78 

43 

77 

School N 

65 

96 

77 

90 

45 

School 0 

41 

98 

89 

91 

27 

School P 

53 

99 

84 

93 

4 

School Q 

44 

91 

94 

39 

3 

Identified by the exceeding achievement expectations method only 

School R 

73 

68 

98 

77 

100 

School S 

82 

57 

95 

88 

98 

School T 

54 

95 

68 

100 

93 

School U 

83 

50 

85 

91 

91 

School V 

85 

35 

34 

96 

87 

School W 

78 

63 

37 

84 

85 

School X 

77 

43 

70 

75 

81 

School Y 

67 

91 

60 

95 

80 

School Z 

75 

79 

58 

85 

75 


Note: The total sample size is 159 schools. The number of students in the cohort by school ranged from 7 to 357. The analyses used 
data on the 2012/13 graduation rate, 2011/12 grade 11 combined reading and math proficiency rate, 2008/09 grade 8 combined 
reading and math proficiency rate, 2009/10 school poverty rate, and data on several other school characteristics for the 2012/13 
school year. 

a. A percentile ranking is the percentage of high-poverty high schools in Puerto Rico that fall at or below a given score. For instance, if a 
school’s performance on an outcome measure was greater than or equal to the performance of 88 percent of the high-poverty schools, 
the percentile ranking for that school would be 88. 

Source: Authors’ analysis based on data from U.S. Department of Education (2010, 2013, 2014) and the Puerto Rico Department of 
Education. 


outcome measure (23 percent; see table A2 in appendix A for details). School poverty rate, 
percentage of students with disabilities, and grade 8 proficiency rate in Spanish were found 
to be statistically significant in both outcome measures. The low to moderate correlations 
between schools’ rankings on individual outcome measures between methods may have 
contributed to the low agreement rate between the two lists of beating-the-odds schools. 
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Table 2. Agreement rate on outcome measures for beating-the-odds high schools in 
Puerto Rico, by identification method, 2011/12 and 2012/13 



Beating-the-odds schools 


Outcome measure 

Criterion 

Identified using 
the status 
method 

Identified using 
the exceeding 
achievement 
expectations 
method 

Identified using 
both 
methods 

Agreement 
rate between 
methods 
(percent) 

Graduation rate, 
2012/13 

Top 

25 percent 

40 

40 

23 

58 

Grade 11 proficiency 
rate (reading and 
math combined), 
2011/12 

Top 

25 percent 

40 

40 

27 

68 

Both measures 

(beating-the-odds 

schools) 

Met all 

criteria 

17 

15 

6 

38 


Note: The total sample size is 159 schools. The number of students in the cohort by school ranged from 7 to 
357. The analyses used data on the 2012/13 graduation rate, 2011/12 grade 11 combined reading and math 
proficiency rate, 2008/09 grade 8 combined reading and math proficiency rate, 2009/10 school poverty rate, 
and data on several other school characteristics for the 2012/13 school year. 

Source: Authors' analysis based on data from U.S. Department of Education (2010, 2013, 2014) and the 
Puerto Rico Department of Education. 


Table 3. Correlations of rankings on outcome measures for beating-the-odds high 
schools in Puerto Rico, by identification methods, 2011/12 and 2012/13 


Ranking by exceeding achievement expectations method 


Graduation rate, 
2012/13 

Combined reading 
and math proficiency 
rate, 2011/12 

Graduation rate, 2012/13 

.76** 

.19* 

Ranking by 

.. . Combined reading and math 

status method b 

proficiency rate, 2011/12 

.20* 

.51** 


* Significant at the .05 level (two-tailed); ** significant at the .01 level (two-tailed). 

Note: The total sample size is 159 schools. The number of students in the cohort by school ranged from 7 to 
357. The analyses used data on the 2012/13 graduation rate, 2011/12 grade 11 combined reading and math 
proficiency rate, 2008/09 grade 8 combined reading and math proficiency rate, 2009/10 school poverty rate, 
and data on several other school characteristics for the 2012/13 school year. 

Source: Authors' analysis based on data from U.S. Department of Education (2010, 2013, 2014) and the 
Puerto Rico Department of Education. 


The characteristics of beating-the-odds schools varied by method 

Qeographic location and locale. Beating-the-odds schools identified using the status 
method were concentrated in the Mayaguez region, with five schools in that region iden- 
tified as beating the odds (table 4). The Bayamon, Humacao, and San Juan regions each 
had three schools identified; the Arecibo, Caguas, and Ponce regions each had one school 
identified. The 15 beating-the-odds schools identified using the exceeding achievement 
expectations method were scattered among six of the seven regions, with the Bayamon, 
Humacao, and Ponce regions each having three and the Arecibo, Mayaguez, and San Juan 
each having two; no school from the Caguas region was identified. 
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Table 4. Region and locale counts of beating-the-odds high schools, by identification 
method, and of all high-poverty high schools in Puerto Rico, 2011/12 and 2012/13 


Region and 



Beating-the-odds schools 


All high-poverty 
high schools in 
Puerto Rico 
(N = 159) 

Identified 
status 
(n = 

using the 
method 

17) 

Identified using 
the exceeding 
achievement 
expectations method 
(n = 15) 

identified using 
both methods 
(n = 6) 

locale 

Number 

Percent 

Number 

Percent 

Number 

Percent 

Number 

Percent 

Region 

Arecibo 

1 

6 

2 

13 

1 

17 

20 

13 

Bayamon 

3 

18 

3 

20 

1 

17 

23 

15 

Caguas 

1 

6 

0 

0 

0 

0 

19 

12 

Humacao 

3 

18 

3 

20 

1 

17 

23 

15 

Mayaguez 

5 

29 

2 

13 

2 

33 

23 

15 

Ponce 

1 

6 

3 

20 

1 

17 

23 

15 

San Juan 

3 

18 

2 

13 

0 

0 

28 

18 

Locale 8 

City 

4 

24 

4 

27 

1 

17 

40 

25 

Suburb 

12 

71 

8 

53 

5 

83 

99 

62 

Town or rural 

1 

6 

3 

20 

0 

0 

20 

13 


a. The 12 National Center for Education Statistics urban-centric locale codes were collapsed into three catego- 
ries for this study. See http://nces.ed.gov/ccd/rural_locales.asp for definitions of the locale codes. 

Source: Authors’ analysis based on data from U.S. Department of Education (2010, 2013, 2014) and the 
Puerto Rico Department of Education. 


Suburban schools had higher representation among beating-the-odds schools identified using 
the status method (71 percent) than among those identified using the exceeding achieve- 
ment expectations method (53 percent). In contrast, for schools located in small towns or 
rural areas a higher percentage of beating-the-odds schools were identified using the exceed- 
ing achievement expectations method (20 percent) than the status method (6 percent). 

School performance. Beating-the-odds schools identified using the status method had 
a higher average graduation rate and a higher average proficiency rate than did beating- 
the-odds schools identified using the exceeding achievement expectations method. The 
average graduation rate in 2012/13 was 84 percent for beating-the-odds schools identified 
using the status method and 77 percent for beating-the-odds schools identified using the 
exceeding achievement expectations method — both higher than the average for all schools 
in the sample (65 percent; figure 1). The average grade 11 proficiency rate was 47 percent 
for beating-the-odds schools identified using the status method and 36 percent for beating- 
the-odds schools identified using the exceeding achievement expectations method, com- 
pared with 24 percent for all schools in the sample. 

Schools that were identified as beating the odds using the status method were ranked lower 
when analyzed by the exceeding achievement expectations method. Under the exceed- 
ing achievement expectations method, when beating-the-odds schools’ observed perfor- 
mance is compared with their expected performance 3 after accounting for students’ prior 
achievement and student demographic characteristics, the average differences (observed 
graduation rate/proficiency rate minus expected graduation rate/proficiency rate) were 
15 percentage points for the graduation rate outcome measure and 16 percentage points for 
the grade 11 proficiency rate outcome measure (figure 2). 


Beating-the-odds 
schools identified 
using the status 
method had a 
higher average 
graduation rate 
and a higher 
average proficiency 
rate than did 
beating-the- 
odds schools 
identified using 
the exceeding 
achievement 
expectations 
method 
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Figure 1. Average observed performance of beating-the-odds high schools in 
Puerto Rico was higher for schools identified using the status method, 2011/12 
and 2012/13 

Percent 


100 -i 


Beating-the-odds schools identified using the status method ( n = 17) 
i Beating-the-odds schools identified using the exceeding achievement expectations method (n = 15) 
Beating-the-odds schools identified using both methods (n = 6) 
i All high-poverty high schools in Puerto Rico ( N = 159) 


75 - 


50 - 



25 - 


Graduation rate, 2012/13 


Grade 11 percent proficient, 2011/12 


Source: Authors' analysis based on data from U.S. Department of Education (2010, 2013, 2014) and the 
Puerto Rico Department of Education. 


Figure 2. Average differences between observed and expected performance 
of beating-the-odds high schools in Puerto Rico were higher for the exceeding 
achievement expectations method, 2011/12 and 2012/13 


Difference (percentage points) 

m Beating-the-odds schools identified using the status method (n = 17) 

20 -| ■ Beating-the-odds schools identified using the exceeding achievement expectations method (n = 15) 
Beating-the-odds schools identified using both methods ( n = 6) 


15 - 


10 - 



Graduation rate, 2012/13 


Grade 11 percent proficient, 2011/12 


Note: The difference between observed and expected performance indicates how much better or worse a 
school actually performed than its characteristics suggest it would. The height of the bars indicates the mean 
difference for each group — that is, on average how much each group of schools performed better than ex- 
pected. By definition, the mean difference between observed and expected performance for all schools in the 
sample is zero under the exceeding achievement expectations method. 

Source: Authors' analysis based on data from U.S. Department of Education (2010, 2013, 2014) and the 
Puerto Rico Department of Education. 
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This difference was to be expected because this method, by definition, identifies schools 
that ranked at the top on differences in performance net of expectations. The average 
difference between observed and expected performance was lower for beating'the'odds 
schools identified using the status method, at 8.8 percentage points for the graduation rate 
measure and 9.3 percentage points for the proficiency rate measure. This suggests that on 
average the schools identified using the status method also performed better than expected 
on the basis of their background characteristics. However, some of the schools (three for 
the graduation rate measure and five for the proficiency rate measure) performed worse 
than expected, with the difference between observed and expected performance being 
as large as 15.4 percentage points for the proficiency rate measure (see table A3 for more 
details). These findings suggest a potential issue with using the status method: looking at 
status alone may identify schools that did not perform better (and potentially performed 
worse) than expected on the basis of their school and student characteristics. 

School characteristics. Beating-the-odds schools identified using the two methods 
appeared similar in average enrollment, student racial/ethnic composition (predominantly 
Hispanic), and student-teacher ratio (table 5). Beating-the-odds schools identified using 
the status method had a slightly higher average percentage of female students than did 
beating'the odds schools identified using the exceeding achievement expectations method. 


Beating-the-odds 
schools identified 
using the two 
methods appeared 
similar in average 
enrollment, 
student racial/ 
ethnic composition, 
and student- 
teacher ratio 


Table 5. Mean characteristics of beating-the-odds high schools, by identification method, and of all 
high-poverty high schools in Puerto Rico, 2011/12 and 2012/13 



Beating-the-odds schools 


School and cohort characteristics 

Identified using 
the status 
method 
(n = 17) 

Identified using 
the exceeding 
achievement 
expectations 
method 
(n = 15) 

Identified using 
both methods 
(n = 6) 

All high-poverty 
high schools in 
Puerto Rico 
(N = 159 a ) 

School characteristic 11 

Total enrollment 

466 

470 

451 

490 

Percentage of female students 

55.9 

51.2 

51.8 

52.1 

Percentage of Hispanic students 

99.7 

99.8 

99.8 

99.8 

Percentage of students from households living below 
the poverty line, 2009/10 

61.3 

74.4 

73.5 

71.2 

Student-teacher ratio 

16.0 

15.1 

15.0 

15.7 

Cohort characteristic 0 

Cohort size 

156 

174 

181 

184 

Percentage of female students 

53.9 

53.0 

51.9 

52.2 

Percentage of Puerto Rican students 0 

97.2 

98.3 

97.6 

98.1 

Percentage of other Hispanic students 0 

2.6 

1.5 

1.9 

1.8 

Percentage of students froom households living below 
the poverty line, 2009/10 

58.3 

66.4 

69.0 

62.9 

Percentage of students with disabilities 

9.6 

17.9 

12.9 

15.6 

Percentage of students with limited Spanish proficiency 

0.5 

0.6 

0.6 

0.5 


a. Sample size may be slightly smaller for some variables because of missing data. 

b. Data are for the 2012/13 school year unless otherwise noted. 

c. For the students who were enrolled in grade 10 in 2010/11 and thus were expected to graduate in 2012/13. 

d. The percentage of students in other racial/ethnic groups is small and not presented here. 

Source: Authors' analysis based on data from U.S. Department of Education (2010, 2013, 2014) and the Puerto Rico Department of 
Education. 
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The average percentage of students from households living below the poverty line was 
higher for beating-the-odds schools identified using the exceeding achievement expecta- 
tions method (74 percent) than for beating-the-odds schools identified using the status 
method (61 percent — lower than the 71 percent for all schools in the sample). 

Cohort characteristics. The 2012/13 graduating cohort in the beating-the-odds schools 
was also similar in average size, student racial and ethnic composition (predominantly His- 
panic), percentage of female students, and percentage of students with limited Spanish 
proficiency. The average percentage of students with disabilities in the 2012/13 graduating 
cohort, however, was lower for beating-the-odds schools identified using the status method 
(10 percent) than for beating-the-odds schools identified using the exceeding achievement 
expectations method (18 percent). Beating-the-odds schools identified using the status 
method also had a lower percentage of students from households living below the poverty 
line in the 2012/13 graduating cohort. 

Implications of the study findings 


This report highlights how identification of beating-the-odds schools can be affected by 
methodological choices. Seventeen schools were identified using the status method, and 15 
schools were identified using the exceeding achievement expectations method; six schools 
were identified using both methods (38 percent agreement rate). The characteristics of 
schools identified as beating-the-odds schools varied depending on which method was 
used and differed from those of all schools in the sample of high-poverty high schools. 
Beating-the-odds schools identified using the status method had a lower average percent- 
age of students from households living below the poverty line and a lower average per- 
centage of students with disabilities than did beating-the-odds schools identified using the 
exceeding achievement expectations method and all schools in the sample. The analyses 
suggest that identifying schools with relatively high percentages of students from house- 
holds living below the poverty line and relatively high performance (the status method) 
is a straightforward method of identifying beating-the-odds schools but that considering 
the exceeding achievement expectations method as well will ensure that schools that are 
doing better than other schools with similarly challenging circumstances are identified. 

The exceeding achievement expectations method adjusts for the potential influence of 
student and school characteristics, including student prior achievement. By statistically 
controlling for these factors, the exceeding achievement expectations method may produce 
less-biased estimates of school performance than does the status method. However, some 
schools identified using the exceeding achievement expectations method have relatively 
low overall graduation rates or relatively low proficiency rates because schools identified 
using this method are by design not as highly ranked. 

The status method identified schools that ranked in the top 25 percent on observed per- 
formance and had at least 40 percent of students from households living below the poverty 
line. However, some of the schools identified using the status method ranked low in per- 
formance when the exceeding achievement expectations method was used (and when 
school poverty rate and other student and school characteristics were statistically con- 
trolled for). These findings suggest that looking at status alone may identify schools that 
did not perform better (and potentially performed worse) than expected on the basis of 
their school and student characteristics. 


It may be useful 
to consider 
both methods of 
identifying beating- 
the-odds schools. 
The high schools 
identified using 
both methods 
demonstrate 
high levels of 
performance 
and appear to be 
achieving higher 
graduation rates 
and proficiency 
rates than might 
be expected given 
their demographics 


10 


Thus, it may be useful to consider both methods of identifying beating-the-odds schools. 
The high schools identified using both methods demonstrate high levels of performance 
and appear to be achieving higher graduation rates and proficiency rates than might be 
expected given their demographics. These schools could suggest the sample of schools 
that could be used in conducting a more in-depth study of organizational structures and 
instructional practices associated with beating-the-odds schools. 

As the work of Abe et al. (2015) suggests, policymakers and researchers must develop a 
rationale for their choices of methods of identifying beating-the-odds schools, with the 
understanding that those choices will influence which schools are ultimately identi- 
fied and, subsequently, what lessons can be drawn from those schools. The limitations 
described in the following section show how challenging it is to identify beating-the-odds 
schools. The value of any lessons learned from future research on beating-the-odds schools 
depends in large part on the precision with which those schools are identified. In this case 
relying solely on either method to identify beating-the-odds schools could confuse policy 
efforts at the Puerto Rico Department of Education to improve low-performing schools. 
That is because the beating-the-odds schools identified either may not be overcoming sig- 
nificant odds (identified using the status method only) or may lack overall high levels of 
achievement (identified using the exceeded achievement expectations method only). 

Limitations of the study 


The study has six main limitations. 

First, the identified schools may not accurately represent beating-the-odds high schools in 
Puerto Rico. The two methods used to identify beating-the-odds schools are based on a 
limited set of data. Because of data limitations, the analyses used data on graduation rates 
for only one cohort and data on their academic achievement in one year. Although school 
performance tends to show stability over time (Dumay, Coe, & Anumendem, 2014), it may 
be affected by cohort- or year-specific factors. Future analyses should include multiple years 
of data. 

Second, the size of the 2012/13 graduating cohort for schools in the sample varied from 7 to 
357, with two schools having fewer than 20 students in the cohort. Aggregated school-level 
performance based on a small number of students may have large unexplained variation 
(noise). The analysis results, therefore, may be driven in part by noise for the schools where 
few students were used to calculate the outcomes. 

Third, the cutpoints were informed by other beating-the-odds studies but were arbitrari- 
ly determined. Sensitivity analyses, conducted using different cutpoints, examined the 
number of schools identified using each method and the agreement between the two 
methods (see appendix B). The analysis suggests that using different cutpoints changes 
the number of schools identified as beating the odds. The agreement rate between the 
two methods ranges from 27 percent to 47 percent as the cutpoint changes. However, the 
number of schools identified using both methods tends to be stable within certain ranges 
of cutpoints. For example, six schools are identified using both methods for the range of 
cutpoints from 21 percent to 31 percent. Using any cutpoint from this range would there- 
fore result in the same six beating-the-odds schools identified using both methods. Using 
25 percent as the cutpoint (instead of 21 percent to 31 percent, for example) puts schools 


The value of any 
lessons learned 
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research on 
beating-the-odds 
schools depends 
in large part on 
the precision 
with which those 
schools are 
identified. In this 
case relying solely 
on either method 
to identify beating- 
the-odds schools 
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policy efforts 
at the Puerto 
Rico Department 
of Education 
to improve 
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schools 
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into commonly used and understood categories (that is, quartiles), and beating-the-odds 
schools were in the top quartile on all outcome measures. 


Fourth, the study uses the three-year cohort graduation rate as one of the outcome mea- 
sures to identify schools. The cohort graduation rate follows each cohort of students 
through high school and is considered a more accurate measure of school completion 
than many other methods of calculating graduation rates (Stetser & Stillwell, 2014). The 
U.S. Department of Education requires states to use this metric when reporting school 
completion. The three-year cohort graduation rate used in this study does not distinguish 
between students who moved from public to private schools and students who left school 
entirely. The difference is important because Puerto Rico has a large share of its students 
in private schools, and some dropout from public schools is accounted for by a net move- 
ment of students from public schools to private schools. 4 This matters because transfer 
to private schools could be a different, but potentially related, signal that students and 
parents are dissatisfied with the public school they left. On the other hand, in such cases 
the transition may indicate only a preference for a private school and not a preference for 
another public school. 

Fifth, the ranking by the exceeding achievement expectations method takes into account 
the potential impact that students’ prior achievement and certain schooUevel demograph- 
ic characteristics might have on school performance (graduation rate and grade 11 profb 
ciency rate on state standardized tests). This method may produce a less-biased estimate 
of school performance than does the status method. But estimates from the exceeding 
achievement expectations method may also be biased if the regression model leaves out 
some unknown variable or variables that are correlated with both the outcome (school 
performance) and some of the control variables included in the regression. The analysis for 
the exceeding achievement expectations method was limited by the data provided on the 
characteristics of Puerto Rico high schools. Having data for a more complete set of school 
factors and having multiple years of data may help address this limitation by improving 
the model specification. Because longitudinal data that link grade 8 students to their high 
schools were not available for Puerto Rico, a proxy for grade 8 achievement of the 2012/13 
cohort was developed using the average grade 8 achievement from a list of possible feeder 
schools for each high school. The feeder schools were identified using information on the 
region, district, school location, and distance between schools. However, the proxy used in 
this report for students’ prior achievement was constructed on the basis of limited data and 
may not reflect the actual feeder patterns for high schools in Puerto Rico (for more infor- 
mation see appendix A). Additional school and community characteristics not examined 
in this study, such as parents’ education level or parent or community involvement, also 
might be important factors affecting school success. 


Policymakers 
may want to take 
advantage of other 
data available on 
Puerto Rico schools 
and use them 
in combination 
with lists such as 
those presented 
here to identify 
best practices 
that drive school 
improvement 


Finally, the particular identification results reported here cannot be considered conclusive. 
Policymakers may want to take advantage of other data available on Puerto Rico schools 
(for example, attendance, student engagement data, and course grades) and use them in 
combination with lists such as those presented here to identify best practices that drive 
school improvement. 
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Appendix A. Details on data and methods as well as additional findings 


Two conceptual methods were used to rank schools: a simple status method and the sta- 
tistically more sophisticated exceeding achievement expectations method (Raudenbush, 
2004). The status method ranks schools by observed performance, which is more straight' 
forward and may be easier to understand. School performance, however, is known to be 
influenced by numerous factors, including student characteristics, prior student achieve- 
ment, school resources, and community factors (over which a school has little control). 
Ranking schools by current performance does not take into account the influence of these 
factors on school performance. Schools identified using this method could demonstrate 
high performance, but they may not be beating the odds because school and student char- 
acteristics could suggest advantageous conditions. 

The exceeding achievement expectations method, which produces a ranking that com- 
pares observed performance to expected performance, tries to account for the influence of 
internal and external factors on performance. This method identifies schools whose per- 
formance exceeds expectations or predictions based on prior achievement and the demo- 
graphic characteristics of schools and students. This study used both methods to rank 
schools and compare how the lists of identified schools differ between the two methods. 
This appendix provides details on the data and analyses presented in the report as well as 
additional findings. 

Data sources 

Regional Educational Laboratory (REL) Northeast & Islands obtained school-level data 
from two sources: the Puerto Rico Department of Education and the Common Core of 
Data maintained by the National Center for Education Statistics (U.S. Department of 
Education, 2010, 2013, 2014). The data elements obtained from each source are presented 
in table Al. 

Sample 

The Puerto Rico Department of Education provided school-level graduation rates for 202 
public high schools within the territory as reported for the 2012/13 school year, the most 
recent year for which graduation data were available at the time of this study. However, the 
2011/12 grade 11 assessment file provided by Puerto Rico contains records for 283 schools, 
and the 2011/12 Common Core of Data file contains records for 196 schools that enrolled 
at least one grade 11 student. All together these files from different sources contain 290 
unique schools that might have been targets for this study. 

The following steps were used to construct the analytic sample for this study: 

• Eighty-eight schools that are in Puerto Rico’s 2011/12 grade 11 assessment file but 
lack data on 2012/13 graduation rates were removed. These schools might have 
been mistakenly included for two reasons: all the values (for proficiency rates) for 
these schools are zero in Puerto Rico’s 2011/12 grade 11 assessment file, and the 
2011/12 Common Core of Data file shows that these schools enrolled no grade 11 
students that year. 

• Twenty-eight vocational schools and two alternative schools were removed from 
the file to focus the study only on regular public schools. 
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Table Al. School-level data elements and sources for identifying beating-the-odds 
high schools in Puerto Rico, 2008/09-2012/13 

1 Data element 

Year 

Source | 

Three-year graduation rate, overall and by subgroup 

2012/13 

Puerto Rico Department of 

Education 

Percentage of grade 11 students proficient in reading 
(Spanish) and math, overall and by subgroup 

2011/12 

Puerto Rico Department of 

Education 

Percentage of grade 8 students proficient in reading 
(Spanish) and math, overall and by subgroup 

2008/09 

Puerto Rico Department of 

Education 

School total enrollment and enrollment by grade 

2008/09, 
2011/12, 
and 2012/13 

U.S. Department of Education 
(2010, 2013, 2014) 

School enrollment by gender, race/ethnicity, and free- 
or reduced-price-lunch status 

2012/13 

U.S. Department of Education 
(2010, 2013, 2014) 

School urban-centric locale codes 3 

2008/09, 
2011/12, 
and 2012/13 

U.S. Department of Education 
(2010, 2013, 2014) 

School location and latitude and longitude 

coordinates 

2008/09, 
2011/12, 
and 2012/13 

U.S. Department of Education 
(2010, 2013, 2014) 

School type 

2008/09, 
2011/12, 
and 2012/13 

U.S. Department of Education 
(2010, 2013, 2014) 

School student-teacher ratio 

2012/13 

U.S. Department of Education 
(2010, 2013, 2014) 


a. The 12 National Center for Education Statistics urban-centric locale codes were collapsed into three catego- 
ries for this study. See http://nces.ed.gov/ccd/rural_locales.asp for definitions of the locale codes. 

Source: Authors' compilation. 


• Five schools were not operational in 2012/13 according to the Common Core of 
Data and were excluded from the analysis. 

The sample was further reduced to include only schools with student poverty rates of at 
least 40 percent. The measure of poverty used in this study is the schoohlevel percentage 
of school-age children (ages 5-17) who were from households living below the poverty line 
(as defined by U.S. Census Bureau, n.d.) in 2009/10, the most recent year for which data 
are available from the Puerto Rico Department of Education. The Census Bureau uses a 
set of income thresholds that vary by family size and composition. This measure is differ- 
ent from another commonly used proxy measure for the concentration of low-income stu- 
dents, the percentage of students eligible for the school lunch program (U.S. Department 
of Education, n.d. a). 

This study used the percentage of students from households living below the poverty line 
because there was more variation in this measure (from 28 percent to 87 percent) than in 
the percentage of students eligible for the federal school lunch program (from 91 percent to 
93 percent) among schools in the sample. However, even with the U.S. Census poverty rate, 
the distribution of schools in the sample concentrates on the high end of the measure. Only 
a small number of schools are represented on the lower end, which indicates that these 
schools might be relatively different from the rest of the schools in socioeconomic charac- 
teristics. The analysis was therefore limited to the sample of schools with at least 40 percent 
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of students from households living below the poverty line so only schools with high poverty 
rates were identified as beating the odds. The final sample included 159 schools. 

Outcome measures 

The study focuses on two outcome measures related to overall performance of high 
schools: school graduation rates and percentage of students scoring proficient on grade 11 
state standardized tests in reading and math. The emphasis of this study is on identifying 
schools that consistently perform well across measures: schools with high performance on 
graduation rates and consistently strong academic performance, given the characteristics 
of the student populations they serve. 

Qraduation rate. For graduation outcomes the study focuses on the three-year school-level 
cohort graduation rates as reported by high schools in Puerto Rico for the 2012/13 school 
year. This graduation rate measures the percentage of students from a given cohort who 
graduated in three years. In calculating graduation rates, the cohort of interest is established 
by identifying the number of students from a given cohort who entered grade 10 for the first 
time three years earlier at a particular school and then adjusting that number by adding 
students who subsequently transferred into the cohort and subtracting any students who 
subsequently transferred out, emigrated to another country, or died. The number of suc- 
cessful graduates (defined as graduating within three years for the measure described here) 
is then presented as a proportion of the larger cohort, as outlined in the following equation: 

3-year cohort graduation rate = number of students in the adjusted cohort who earned 
a regular high school diploma by August 2013/Number of first-time grade 10 students in 
2010/11, adjusted for transfers in and out, emigration to another country, and death x 100 

The reported graduation rates for 2012/13 thus reflect the percentages of students who 
entered grade 10 in the 2010/11 school year and graduated at the end of the 2012/13 school 
year. 

Qrade 11 proficiency rates in reading and math in 2011/12. Student achievement out- 
comes included data on grade 11 proficiency rates in reading (in Spanish, the language of 
instruction in Puerto Rico) and math in the 2011/12 school year, provided by the Puerto 
Rico Department of Education. Because Puerto Rico tests students when they are in grade 
11, the assessment results reported for the 2011/12 school year reflect the performance of 
the cohort of students who entered grade 10 in 2010/11 and were in grade 11 when tested 
— the same cohort that is reflected in the 2012/13 graduation data. The study team cal- 
culated an average proficiency rate for reading and math for each school weighted by the 
number of students tested in each subject and used the weighted average as an outcome 
measure in the analysis. 

Additional notes on ranking schools using the exceeding achievement expectations method 

For the exceeding achievement expectations method, the expected performance of each 
high school was calculated, adjusting for students’ prior achievement and various other 
student and school characteristics. Then each school’s observed performance was com- 
pared with its expected performance and schools were ranked on the basis of the extent to 
which their observed performance exceeded (or fell short of) their expected performance. 
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Constructing a proxy for measuring students’ prior achievement. The best predictor of 
students’ academic performance is their past performance (Ingels, Curtin, Kaufman, Alt, 
& Chen, 2002). However, the study team did not have access to data that matched Puerto 
Rico high school students to their grade 8 scores. Consequently, there was not sufficient 
data to support a longitudinal analysis that controlled for students’ prior achievement in 
order to rank high schools by exceeding achievement expectations. Grade 8 school means 
on the state standardized test could not be directly linked to high school (grade 11) test 
means because school feeder patterns were not known with sufficient certainty. However, 
possible feeder schools were identified for each high school in the sample using information 
on region, 5 district, 6 school location (city and ZIP code), and distance between schools. For 
this analysis, data from the 2008/09 school year were used because that is the year the 
targeted cohort of students (those who entered grade 10 in the 2010/11 school year) were 
in grade 8, the pre-high school grade that was tested in state standardized exams. For 
each high school the following steps were used to identify a set of schools with grade 8 
students enrolled and tested in 2008/09 that matched on as many of these characteristics 
as possible: 

1. A total of 403 regular public schools were identified that enrolled at least one grade 8 
student according to the 2008/09 Common Core of Data (U.S. Department of Educa- 
tion, 2010) and that reported percentage proficient in reading or math in the Puerto 
Rico 2008/09 assessment data (hereafter referred to, for simplicity, as middle schools). 

2. The distance between each pair of middle schools and high schools was calculated 
using school latitude and longitude coordinates available from the Common Core of 
Data. 7 

3. Both the middle schools and the high schools were sorted by region, district, and ZIP 
code. 

4- Middle schools and high schools were matched by region and district. If there was only 
one high school in a district, all the middle schools in that district were matched to 
that high school (with the exceptions described in step 6). 

5. If there were multiple high schools in a district, middle schools were matched to high 
schools by taking into account the distance between schools and the cohort size. 
Each middle school would first be matched to the closest high school. However, if the 
number of grade 8 students in 2008/09 matched to a high school was considerably 
larger (for example, two times larger) than the size of the grade 11 cohort in 2011/12, 
adjustments were made so that some middle schools originally matched to that high 
school would be matched to the next closest high school in which the number of 
matched grade 8 students had not reached the size limit. Three middle schools were 
matched using this approach. 

6. For three middle schools the distances between them and the high schools in their 
districts were unusually large (for example, more than 12 miles), whereas the distance 
between these middle schools and a high school in a nearby district was rather small 
(for example, less than 3 miles). These middle schools were matched to the high school 
in the nearby district instead of a high school in their own district. 
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Using this approach, the study matched 403 middle schools with the 159 high schools in 
the sample. The number of middle schools matched to each high school ranged from one 
to six, with an average of 2.4 schools. For each high school, the study team calculated an 
average of schooUevel grade 8 proficiency rate for its matched middle schools, weighted by 
number of grade 8 students tested in each school. The weighted averages (one for reading 
and one for math) were then used as proxies for students’ prior achievement for each high 
school. 

Comparing observed performance with expected performance. A school’s expected pen 
formance was calculated given students’ prior achievement, socioeconomic background, 
and other school characteristics, such as region, locale, school size, percentage of female 
students, percentage of students with disabilities, and percentage of students with limited 
Spanish proficiency. An example of this approach using graduation rate as the outcome 
measure follows. First, an expected graduation rate for each high school was estimated 
given students’ prior achievement and characteristics of its student population. This was 
accomplished by regressing (using ordinary least squares) the outcome measure (2012/13 
graduation rate) on a set of predictors, including the weighted averages of grade 8 mean 
proficiency rate in reading and math and other schooUevel characteristics of students and 
school characteristics (table A2). This regression analysis produced a set of coefficients 
associated with each predictor (or variable) in the model, indicating the relationship 
between each predictor and the graduation outcome. The coefficients were then used to 
compute an expected graduation rate for each high school, given its various characteris' 
tics. The difference between the observed and expected rates indicates how much better 
or worse a school performed on graduating students than its characteristics suggested it 
would. This difference (performance net of expectations) was then used to rank schools. 

The same procedures were used to estimate how much better or worse a school performed 
on the proficiency rate measure given the school’s characteristics. The differences between 
observed and expected performance were used to rank schools for each outcome measure. 

Comparing and describing characteristics of beating-the-odds schools 

The study team compared the lists of beating-the-odds schools identified using the two 
methods and computed an agreement rate between the two lists. This agreement rate 
measures the share of commonly identified schools between two beating-the-odds lists. 
The agreement rate ranges from 0 percent (if the two lists share no commonly identified 
schools) to 100 percent (if the two lists have exactly the same schools). 

Finally, the characteristics of the two lists of beating-the-odds schools were compared with 
all high schools in the sample. The report contains basic descriptive information (frequen- 
cy distribution for categorical variables and minimum, maximum, mean, and standard 
deviation for continuous variables) for the graduation rate outcome, achievement outcome, 
student demographic characteristics, school locale, enrollment, number of full-time equiv- 
alent teachers, and student-teacher ratio (see tables 4 and 5 in the main text and tables 
A3 and A4). 


Table A2. Predictors used in regression models and regression results for Puerto 
Rico high-poverty high schools, by geographic region and locale, 2011/12 and 
2012/13 


Predictor 

Graduation rate, 
2012/13 
(N = 159) 

Combined reading and math 
proficiency rate, 2011/12 
(N = 159) 

Coefficient 

p-value 

Coefficient 

p-value 

Region 8 

Arecibo 

0.59 

.91 

-0.10 

.98 

Bayamon 

2.38 

.63 

0.67 

.89 

Caguas 

3.65 

.42 

3.55 

.42 

Humacao 

0.72 

.88 

-8.46 

.08 

Mayaguez 

2.93 

.53 

5.99 

.20 

Ponce 

3.84 

.40 

7.06 

.12 

Locale 8 

City 

-4.53 

.30 

-5.08 

.23 

Suburb 

2.94 

.38 

1.72 

.60 

Other predictors 

Size of grade 11 cohort 

0.00 

.92 

-0.03 

.05 

School percentage of female students 

0.02 

.91 

0.31 

.14 

School percentage of students from 





households living below the poverty line 

-0.44** 

.00 

-0.63** 

.00 

Cohort-level percentage of students 





with disabilities 8 

-0.38* 

.01 

-0.35* 

.02 

Cohort-level percentage of students 





with limited Spanish proficiency 8 

-0.45 

.71 

0.66 

.58 

Grade 8 mean proficiency rate in math 

0.13 

.40 

-0.03 

.84 

Grade 8 mean proficiency rate in 





reading (Spanish) 

0.20* 

.02 

0.26** 

.00 

Adjusted R 2 

.23 



.36 


* Significant at the .05 level (two-tailed); ** significant at the .01 level (two-tailed). 

Note: The total sample size is 159 schools. The number of students in the cohort by school ranged from 7 to 
357. The analyses used data on the 2012/13 graduation rate, 2011/12 grade 11 combined reading and math 
proficiency rate, 2008/09 grade 8 combined reading and math proficiency rate, 2009/10 school poverty rate, 
and data on several other school characteristics for the 2012/13 school year. 

a. San Juan is the reference category. 

b. The 12 National Center for Education Statistics urban-centric locale codes were collapsed into three catego- 
ries for this study. See http://nces.ed.gov/ccd/rural_locales.asp for definitions of the locale codes. 

c. Data are available at the cohort level only. 

Source: Authors' analysis based on data from U.S. Department of Education (2010, 2013, 2014) and the 
Puerto Rico Department of Education. 
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Table A3. Performance of beating-the-odds high schools in Puerto Rico, by identification method and outcome measure, and of all 
high-poverty high schools in Puerto Rico, 2011/12 and 2012/13 







Beating-the-odds-schools 











Identified using 
the status method 
(n = 17) 

Identified using 
the exceeding achievement 
expectations method 
(n = 15) 


Identified using 
both methods 
(n = 6) 



All schools in sample 
(N = 159) 

Outcome measure 

Mini- 

mum 

Maxi- 

mum 

Mean 

Standard 

deviation 

Mini- 

mum 

Maxi- 

mum 

Mean 

Standard 

deviation 

Mini- 

mum 

Maxi- 

mum 

Mean 

Standard 

deviation 

Mini- 

mum 

Maxi- 

mum 

Mean 

Standard 

deviation 

Observed performance (percent) 

Graduation rate, 2012/13 

74.7 

100.0 

83.7 

7.2 

63.2 

90.6 

76.5 

8.8 

75.6 

90.6 

83.1 

6.5 

0.0 

100.0 

65.2 

14.4 

Grade 11 proficiency rate 
(reading and math combined), 
2011/12 

27.0 

94.0 

46.5 

20.1 

17.0 

80.0 

35.6 

17.3 

27.0 

59.0 

40.3 

11.4 

0.0 

97.0 

24.1 

15.4 

Difference between observed and expected performance (percentage points) 

Graduation rate, 2012/13 

-4.6 

31.1 

8.8 

8.7 

7.8 

44.7 

15.1 

10.1 

8.6 

31.1 

16.3 

8.4 

-58.6 

44.7 

0.0 

12.0 

Grade 11 proficiency rate 
(reading and math combined), 
2011/12 

-15.4 

34.8 

9.3 

15.0 

3.0 

57.8 

16.4 

14.7 

5.3 

27.5 

14.9 

9.2 

-22.1 

57.8 

0.0 

11.7 

Source: Authors' analysis based on data from U.S. 

Department of Education (2010, 2013, 2014) and the Puerto Rico Department of Education. 
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Table A4. Characteristics of beating-the-odds high schools in Puerto Rico, by identification method, and of all high-poverty high schools 
in Puerto Rico, 2011/12 and 2012/13 




Identified using 
the status method 
(n = 17) 

Identified using 
the exceeding achievement 
expectations method 
(n = 15) 


Identified using 
both methods 
(n = 6) 


All schools in sample 
(N = 159) 

Outcome measure 

Mini- 

mum 

Maxi- 

mum 

Mean 

Standard 

deviation 

Mini- 

mum 

Maxi- 

mum 

Mean 

Standard 

deviation 

Mini- 

mum 

Maxi- 

mum 

Mean 

Standard 

deviation 

Mini- 

mum 

Maxi- 

mum 

Mean 

Standard 

deviation 

School characteristic® 

Total enrollment 

126 

823 

466 

225 

127 

750 

470 

200 

152 

750 

451 

232 

98 

934 

490 

192 

Percentage of female students 

46.9 

95.2 

55.9 

11.0 

45.7 

55.7 

51.2 

3.1 

46.9 

55.7 

51.8 

3.3 

38.6 

95.2 

52.1 

5.4 

Percentage of Hispanic students 

98.1 

100.0 

99.7 

0.5 

98.4 

100.0 

99.8 

0.4 

99.3 

100.0 

99.8 

0.2 

97.8 

100.0 

99.8 

0.4 

Percentage of students from 
households living below the 
poverty line, 2009/10 

41.3 

82.1 

61.3 

13.3 

54.2 

85.4 

74.4 

8.4 

64.8 

82.1 

73.5 

6.9 

5.0 

23.0 

15.5 

3.3 

Student-teacher ratio 

7.0 

21.0 

16.0 

4.1 

7.0 

20.0 

15.1 

3.8 

64.8 

82.1 

73.5 

6.9 

8.0 

19.0 

15.0 

4.2 

Cohort characteristic 11 

Cohort size 

10 

357 

156 

112 

7 

357 

174 

100 

32 

357 

181 

115 

7 

357 

184 

89 

Percentage of female students 

37.0 

90.0 

53.9 

11.9 

44.9 

71.4 

53.0 

7.4 

44.9 

62.5 

51.9 

6.9 

22.9 

90.0 

52.2 

6.9 

Percentage of Puerto Rican 

students 0 

85.2 

100.0 

97.2 

3.8 

93.9 

100.0 

98.3 

2.1 

93.9 

100.0 

97.6 

2.5 

81.1 

100.0 

98.1 

2.8 

Percentage of other Hispanic 

students 0 

0.0 

14.8 

2.6 

3.9 

0.0 

6.1 

1.5 

2.1 

0.0 

6.1 

1.9 

2.6 

0.0 

19.0 

1.8 

2.8 

Percentage of students from 
households living below the 
poverty line 

17.7 

82.1 

58.3 

16.8 

9.9 

85.5 

66.4 

18.5 

56.8 

82.1 

69.0 

9.9 

2.9 

90.1 

62.9 

17.1 

Percentage of students with 

disabilities 

0.0 

19.6 

9.6 

5.7 

0.0 

71.4 

17.9 

15.7 

0.0 

19.6 

12.9 

7.0 

0.0 

71.4 

15.6 

7.8 

Percentage of students with 
limited Spanish proficiency 

0.0 

3.2 

0.5 

0.8 

0.0 

1.6 

0.6 

0.6 

0.0 

1.4 

0.6 

0.6 

0.0 

6.7 

0.5 

0.9 


a. Data are for the 2012/13 school year unless otherwise noted. 

b. Data are for the cohort of students who entered grade 10 in 2010/11 and were expected to graduate in 2012/13. 

c. The percentage of students in other racial/ethnic groups is small and not presented here. 

Source: Authors’ analysis based on data from U.S. Department of Education (2010, 2013, 2014) and the Puerto Rico Department of Education. 





Appendix B. Sensitivity analysis results 


This appendix presents the results from the sensitivity analysis that was conducted using 
different outpoints for the outcome measures. The analysis shows that as the outpoint 
expanded from the top 10 percent of schools to include the top 40 percent, the number of 
beating-the-odds high schools identified expanded from 8 to 33 for the status method and 
from 5 to 32 for the exceeding achievement expectations method, and the agreement rate 
between the two methods fluctuated between 27 percent and 47 percent (table Bl). 


Table Bl. Number of beating-the-odds high schools in Puerto Rico identified using 
different cutpoints for outcome measures, by identification method, 2011/12 and 
2012/13 


Cutpoint 

(percent) 

Beating-the-odds schools ( N = 159) 



Identified using the 
status method 

Identified using the 
exceeding achievement 
expectations method 

Identified using 
both methods 

Agreement 

rate 

(percent) 

10 

8 

5 

3 

46 

11 

9 

5 

3 

43 

12 

9 

6 

3 

40 

13 

10 

8 

3 

33 

14 

11 

8 

3 

32 

15 

11 

8 

3 

32 

16 

12 

8 

3 

30 

17 

12 

9 

3 

29 

18 

13 

9 

3 

27 

19 

14 

10 

4 

33 

20 

14 

11 

4 

32 

21 

15 

12 

6 

44 

22 

16 

12 

6 

43 

23 

16 

12 

6 

43 

24 

17 

13 

6 

40 

25 

17 

15 

6 

38 

26 

18 

16 

6 

35 

27 

19 

16 

6 

34 

28 

20 

17 

6 

32 

29 

20 

18 

6 

32 

30 

21 

18 

6 

31 

31 

22 

20 

6 

29 

32 

22 

21 

7 

33 

33 

24 

22 

9 

39 

34 

24 

25 

11 

45 

35 

25 

26 

12 

47 

36 

26 

26 

12 

46 

37 

28 

27 

12 

44 

38 

30 

29 

13 

44 

39 

31 

30 

14 

46 

40 

33 

32 

15 

46 


Note: The analyses used data on the 2012/13 graduation rate, 2011/12 grade 11 reading and math profi- 
ciency rate, 2008/09 grade 8 reading and math proficiency rate, 2009/10 school poverty rate, and data on 
several other school characteristics for the 2012/13 school year. 

Source: Authors’ analysis based on data from U.S. Department of Education (2010, 2013, 2014) and the 
Puerto Rico Department of Education. 
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Notes 


1. These schools ranked in the 76th percentile or higher on each measure for the status 
method, meaning that they performed better than at least 75 percent of schools on a 
specific measure. 

2. There is no absolute guide for the magnitude of correlation coefficients that indicates 
when two variables have a low or high degree of correlation. Hinkle, Wiersma, and 
Jurs (2003) suggest interpreting a correlation coefficient of .90 to 1.00 (-.90 to -1.00) as 
a very high positive (negative) correlation, a correlation coefficient of .70 to .90 (-.70 
to -.90) as a high positive (negative) correlation, a correlation coefficient of .50 to .70 
(-.50 to -.70) as a moderate positive (negative) correlation, a correlation coefficient 
of .30 to .50 (-.30 to -.50) as a low positive (negative) correlation, and a correlation 
coefficient of .0 to .30 (.0 to -.30) as little if any correlation. 

3. The expected performance for a school indicates the average performance for a school 
with a particular set of characteristics that are included as variables in the regression 
model. The difference between the observed and expected performance indicates how 
much better or worse a school actually performed than its characteristics predicted it 
would. A positive difference indicates that the school performed better than expect- 
ed, given its characteristics. A negative difference indicates that the school performed 
worse than expected. 

4. According to the U.S. Census Bureau’s 2012 American Community Survey, 77 percent 
of all kindergarten to grade 12 students in Puerto Rico attended public schools. 

5. The seven regions identified in the data are Arecibo, Bayamon, Caguas, Humacao, 
Mayaguez, Ponce, and San Juan. 

6. It is not clear how districts are defined in the data provided by the Puerto Rico Depart- 
ment of Education. By comparing data provided by Puerto Rico and the Common 
Core of Data, it appears that “districts” in the files provided by Puerto Rico largely 
overlap with municipalities or counties in the Common Core of Data. For U.S. Census 
purposes, the municipalities are equivalent to counties. Puerto Rico has 78 municipal- 
ities or “municipios.” 

7. The study team used the haversine formula to calculate the distance between two 
schools. This formula gives the great-circle distances between two points on a sphere 
from their longitudes and latitudes. 
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